Cognitive Principles in Multimodal Interpretation

Multimodal conversational interfaces provide a natural means for users to communicate with computer systems through multiple modalities such as speech and gesture. To build effective multimodal interfaces, understanding user multimodal inputs is important. Previous studies have shown that user language behavior does not occur randomly, but rather follows certain linguistic and cognitive principles. Therefore, this paper investigates the use of cognitive principles in multimodal input interpretation. In particular, we address how to incorporate the ideas of Conversation Implicature and Givenness Hierarchy in a greedy algorithm for efficient multimodal reference resolution. Our empirical studies indicate that this algorithm, while maintaining the same performance, significantly reduces the complexity in searching for a solution compared to a previous graph-matching approach. One major advantage of this greedy algorithm is that the prior linguistic and cognitive knowledge can be used to guide the search process and significantly prune the search space. Because of its simplicity and generality, this approach has the potential to improve the robustness of interpretation and provide a more practical solution to multimodal input interpretation.